System and method for determining disk failure indicator to predict future disk failures

ABSTRACT

Techniques for determining a disk failure indicator for predicting disk failures are described herein. According to one embodiment, diagnostic parameters are received which are collected from a set of known working disks and a set of known failed disks of a storage system. For each of the diagnostic parameters, a first quantile distribution representation is generated for the set of known working disks, and a second quantile distribution representation is generated for the set of known failed disks. The first quantile distribution representation and the second quantile distribution representation of each of the diagnostic parameters are then compared to select one or more of the diagnostic parameters as one or more disk failure indicators for predicting future disk failures.

RELATED APPLICATIONS

This application is related to U.S. patent application Ser. No.14/037,202, entitled “System and Method for Predicting Single-DiskFailures,” filed Sep. 25, 2013, and U.S. patent application Ser. No.14/037,204, entitled “System and Method for Predicting Multiple-DiskFailures,” filed Sep. 25, 2013. The disclosure of the above applicationsis incorporated by reference herein in its entirety.

FIELD OF THE INVENTION

Embodiments of the present invention relate generally to data storagesystems. More particularly, embodiments of the invention relate todetermining a disk failure indicator for predicting future diskfailures.

BACKGROUND

Data storage utilization is continually increasing, causing theproliferation of storage systems in data centers. Monitoring andmanaging these systems require increasing amounts of human resources.Information technology (IT) organizations often operate reactively,taking action only when systems reach capacity or fail, at which pointperformance degradation or failure has already occurred. Hard diskfailures fall into one of two basic classes: predictable failures andunpredictable failures. Predictable failures result from slow processessuch as mechanical wear and gradual degradation of storage surfaces.Monitoring can determine when such failures are becoming more likely.Unpredictable failures happen suddenly and without warning. They rangefrom electronic components becoming defective to a sudden mechanicalfailure (perhaps due to improper handling).

Self-Monitoring, Analysis and Reporting Technology (S.M.A.R.T., orsimply written as SMART) is a monitoring system for computer hard diskdrives to detect and report on various indicators of reliability, in thehope of anticipating failures. When a failure is anticipated byS.M.A.R.T., the user may choose to replace the drive to avoid unexpectedoutage and data loss. The manufacturer may be able to use the S.M.A.R.T.data to discover where faults lie and prevent them from recurring infuture drive designs. However, not all of the S.M.A.R.T. attributes canconsistently provide reliable indications of possible disk failures. TheS.M.A.R.T. attributes tend to vary and they may have differentinterpretation from one hard disk vendor or configuration to another.There has been a lack of reliable mechanism to determine which of theS.M.A.R.T. attributes to be the best disk failure indicator, as well asthe efficient ways to predict single disk failure or multi-disk failuresin a redundant array independent disks (RAID) environment.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are illustrated by way of example and notlimitation in the figures of the accompanying drawings in which likereferences indicate similar elements.

FIG. 1 is a block diagram illustrating a storage system according to oneembodiment of the invention.

FIG. 2 is a flow diagram illustrating a method for predicting diskfailures according to one embodiment of the invention.

FIGS. 3A and 3B are diagrams illustrating certain quantile distributionrepresentations which may be used with an embodiment of the invention.

FIG. 4 is a flow diagram illustrating a method for determining a diskfailure indicator according to one embodiment of the invention.

FIG. 5 is a flow diagram illustrating a method for predicting diskfailures of a single disk according to one embodiment of the invention.

FIG. 6 is a diagram illustrating a process for determining an optimalthreshold for predicting disk failures according to one embodiment ofthe invention.

FIG. 7 is a flow diagram illustrating a method for determining anoptimal threshold for predicting disk failures according to oneembodiment of the invention.

FIGS. 8A and 8B are diagrams illustrating a process for predictingmulti-disk failures according to one embodiment of the invention.

FIG. 9 is a flow diagram illustrating a method for predicting multipledisk failures according to one embodiment of the invention.

FIG. 10 is a block diagram illustrating a deduplicated storage systemaccording to one embodiment of the invention.

DETAILED DESCRIPTION

Various embodiments and aspects of the inventions will be described withreference to details discussed below, and the accompanying drawings willillustrate the various embodiments. The following description anddrawings are illustrative of the invention and are not to be construedas limiting the invention. Numerous specific details are described toprovide a thorough understanding of various embodiments of the presentinvention. However, in certain instances, well-known or conventionaldetails are not described in order to provide a concise discussion ofembodiments of the present inventions.

Reference in the specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin conjunction with the embodiment can be included in at least oneembodiment of the invention. The appearances of the phrase “in oneembodiment” in various places in the specification do not necessarilyall refer to the same embodiment.

According to one aspect of the invention, a method using quantiledistribution techniques is provided to select one or more of diagnosticparameters, such as S.M.A.R.T. attributes and/or small computer systeminterface (SCSI) disk return codes, that are collected from the disks tobe the most reliable disk failure indicator or indicators for a givencertain storage configuration or environment (e.g., target storagesystem or disks). According to one embodiment, diagnostic parameters arecollected from a set of historically known failed disks and knownworking disks of a particular storage system or configuration (e.g., astorage system or systems deployed at a particular enterprise ororganization). The diagnostic parameters of the known failed disks andknown working disks are analyzed or trained in view each other todetermine which of the those diagnostic parameters can be used to mostefficiently or consistently distinguish or identify a failed disk orworking disk from the set of disks.

According to one embodiment, for each of the diagnostic parameters, afirst quantile distribution representation and a second quantiledistribution representation (e.g., quantile distribution graphs orcurves) are generated for the known failed disks and the known workdisks, respectively. For each of the diagnostic parameters, the firstand second quantile distribution representations are analyzed todetermine its maximum difference value between the first and secondquantile distribution representations. The maximum difference values ofall the diagnostic parameters are then compared to each other to selectone or more of the diagnostic parameters that are largest amongst all orabove certain predetermined threshold as one or more disk failureindicators, which can consistently identify a potential failed disk froma group of disks.

According to another aspect of the invention, the selected disk failureindicator may be used to predict the disk failure probability of aparticular disk. According to one embodiment, values of a predetermineddiagnostic parameter (representing the selected disk failure indicator,such as the reallocated sector count as part of S.M.A.R.T. attributes orthe medium error as part of SCSI disk return codes) are collected from aset of known failed disks and known working disks associated with atarget storage system. A first and second quantile distribution graphsof the values of the predetermined diagnostic parameter for the knownfailed disks and known working disks are generated, respectively, inview of a range of values of the known failed disks. An optimalthreshold of the values of the predetermined diagnostic parameter isdetermined by comparing the first and second quantile distributiongraphs. In one embodiment, the optimal threshold is identified at aposition where the difference between the first and second quantiledistribution graphs reaches the maximum. The optimal threshold is thenutilized to indicate or predict whether a particular target disk mayhave a higher probability of disk failure, for example, by comparing thevalue of the predetermined diagnostic obtained from the target diskagainst the optimal threshold.

According to another aspect of the invention, the selected disk failureindicator may be used to predict the disk failure probability ofmultiple disks in a RAID configuration. According to one embodiment,values of a predetermined diagnostic parameter (representing theselected disk failure indicator, such as the reallocated sector count aspart of S.M.A.R.T. attributes) are collected from a set of known faileddisks associated with a target storage system. A quantile distributiongraph is generated for the values in view of percentiles (e.g., 10%,20%, . . . , 100%) of a number of known failed disks involved in thequantile distribution. Subsequently, when values of the predetermineddiagnostic parameter are collected from a set of target storage disks,the collected values are applied to the quantile distribution graph todetermine their respective percentiles of the target disks. Eachpercentile represents a probability of the corresponding disk failure.The individual disk failure probabilities are then used to calculate theprobability of disk failures of two or more of the target disks.

FIG. 1 is a block diagram illustrating a storage system according to oneembodiment of the invention. Referring to FIG. 1, system 100 includes,but is not limited to, one or more client systems 101-102communicatively coupled to one or more storage systems 104 over network103. Clients 101-102 may be any type of clients such as a server, apersonal computer (e.g., desktops, laptops, and tablets), a “thin”client, a personal digital assistant (PDA), a Web enabled appliance, agaming device, a media player, or a mobile phone (e.g., Smartphone),etc. Network 103 may be any type of networks such as a local areanetwork (LAN), a wide area network (WAN) such as Internet, or acombination thereof.

Storage system 104 may include any type of server or cluster of servers.For example, storage system 104 may be a storage server used for any ofvarious different purposes, such as to provide multiple users withaccess to shared data and/or to back up mission critical data. In oneembodiment, storage system 104 includes, but is not limited to, backupengine 106, optional deduplication storage engine 107, and one or morestorage units 108-109 communicatively coupled to each other. Storageunits 108-109 may be implemented locally (e.g., single node operatingenvironment) or remotely (e.g., multi-node operating environment) viainterconnect 120, which may be a bus and/or a network.

In response to a data file to be stored in storage units 108-109,deduplication storage engine 107 is configured to segment the data fileinto multiple chunks (also referred to as segments) according to avariety of segmentation policies or rules. Deduplication storage engine107 may choose not to store a chunk in a storage unit if the chunk hasbeen previously stored in the storage unit. In the event thatdeduplication storage engine 107 chooses not to store the chunk in thestorage unit, it stores metadata enabling the reconstruction of the fileusing the previously stored chunk. As a result, chunks of data files arestored in a deduplicated manner, either within each of storage units108-109 or across at least some of storage units 108-109. The metadata,such as metadata 110-111, may be stored in at least some of storageunits 108-109, such that files can be accessed independent of anotherstorage unit. Metadata of each storage unit includes enough informationto provide access to the files it contains.

According to one embodiment, storage system 104 further includes anoperation manager 105 to manage and monitor operations performed bystorage system 104, including periodically collecting and transmittingoperating diagnostic data to a remote device such as management system150 over network 103. In this example as shown in FIG. 1, storage system104 may be located at a client site and utilized by a client such as anenterprise or corporation, where the storage system 104 may be providedby a storage provider or vendor such as EMC® Corporation. In oneembodiment, operation manager 105 periodically collects operatingstatistics concerning operations of storage units 108-109 and transmitsdiagnostic data representing at least some of the operating statisticsto management system 150, where management system 150 is associated witha storage provider or vendor that provides storage system 104 to aclient. For example, management system 150 may be operated or owned bythe storage provider or alternatively, it may be operated by athird-party vendor on behalf of the storage provider. In one embodiment,the diagnostic data may include diagnostic parameters such as thosedefined by S.M.A.R.T. specification and/or those defined as part of theSCSI disk return codes, which may be collected from the storage system104. For example, operation manager 105 may include or communicate witha S.M.A.R.T. tool or software configured to monitor operations ofstorage units 108-109. Each of the storage units 108-109 may beimplemented one or more individual disks or alternatively, a RAID arrayof disks.

Note that storage system 104 may represent a group or cluster ofindividual storage systems, where operation manager 105 of each storagesystem may be equipped with a “phone-home” functionality that mayperiodically transmit operating status of the respective storage system,including the diagnostic parameters (e.g., S.M.A.R.T. attributes andSCSI return codes) of the associated storage disks, to a centralized ordistributed entity, such as management server 150 or dedicated datacollection entity 160 (e.g., a third-party data collection agent).

According to one embodiment, management system 150 includes a datacollector 151, disk failure predictor 152, and analysis module 153. Datacollector 151 is employed to communicate with operation manager 105 ofstorage system(s) 104 to collect diagnostic data concerning operatingstatuses of storage units 108-109, as well as storage system 104 ingeneral. Note that although one storage system is shown in FIG. 1, datacollector 151 may communicate with multiple operation managers ofmultiple storage systems to collect diagnostic data concerning therespective storage systems, which may be located at the same ordifferent geographical locations (e.g., same or different client sites).For example, management system 150 may be a centralized managementserver or cluster of servers (e.g., in the cloud) for single or multipleclients or customers.

The collected diagnostic data is stored in a storage device as part ofdiagnostic logs 154. In one embodiment, diagnostic data 154 includesdiagnostic parameters collected from various storage systems such asstorage system 104. The diagnostic parameters may be those attributes(e.g., reallocated sector, pending sector, uncorrectable sector, etc.)defined by S.M.A.R.T. as shown in the Appendix I below. Alternatively,diagnostic parameters may be those from the SCSI return codes (e.g.,medium error, timeout, connection error, data error, etc.) as shown inthe Appendix II below. In one embodiment, analysis module 153 is toperform an analysis on the diagnostic data 154 such as determining whichof the diagnostic parameters can be used as the best disk failureindicator(s). Disk failure predictor 152 is configured to predict, usingthe disk failure indicator(s), which one or more of the disks of storageunits 108-109 of storage system 104 have a higher probability of diskfailures.

According to one embodiment of the invention, analysis module 153utilizes quantile distribution techniques to select one or more ofdiagnostic parameters, such as S.M.A.R.T. attributes, SCSI return codes,that are collected from the disks of storage units 108-109 to be themost reliable disk failure indicator or indicators for a given certainstorage configuration or environment (e.g., target storage system ordisks). According to one embodiment, diagnostic parameters arecollected, for example, by data collector 151 or server 160, from a setof historically known failed disks and known working disks of aparticular storage system or configuration (e.g., a storage system orsystems deployed at a particular enterprise or organization). Thediagnostic parameters of the known failed disks and known working disksare analyzed or trained by analysis module 153 in view each other todetermine which of the those diagnostic parameters can be used to mostefficiently or consistently distinguish or identify a failed disk orworking disk from the set of disks.

According to one embodiment, for each of the diagnostic parameters, afirst quantile distribution representation and a second quantiledistribution representation (e.g., quantile distribution graphs orcurves) are generated for the known failed disks and the known workdisks, respectively. For each of the diagnostic parameters, the firstand second quantile distribution representations are analyzed todetermine its maximum difference value between the first and secondquantile distribution representations. The maximum difference values ofall the diagnostic parameters are then compared to each other to selectone or more of the diagnostic parameters that are largest amongst all orabove certain predetermined threshold as one or more disk failureindicators, which can consistently identify a potential failed disk froma group of disks.

According to another embodiment of the invention, the selected diskfailure indicator may be used by disk failure predictor 152 to predictthe disk failure probability of a particular disk. According to oneembodiment, values of a predetermined diagnostic parameter (representingthe selected disk failure indicator, such as the reallocated sectorcount as part of S.M.A.R.T. attributes) are collected (e.g., bycollector 151 or server 160) from a set of known failed disks and knownworking disks associated with a target storage system. A first andsecond quantile distribution graphs of the values of the predetermineddiagnostic parameter for the known failed disks and known working disksare generated, respectively, in view of a range of values of the knownfailed disks. An optimal threshold of the value of the predetermineddiagnostic parameter is determined by comparing the first and secondquantile distribution graphs. In one embodiment, the optimal thresholdis identified at a position where the difference between the first andsecond quantile distribution graphs reaches the maximum. The optimalthreshold is then utilized by disk failure predictor 152 to indicate orpredict whether a particular target disk may have a higher probabilityof disk failure, for example, by comparing the value of thepredetermined diagnostic obtained from the target disk against theoptimal threshold.

According to a further embodiment of the invention, the selected diskfailure indicator may be used to predict the disk failure probability ofmultiple disks in a RAID configuration. According to one embodiment,values of a predetermined diagnostic parameter (representing theselected disk failure indicator, such as the reallocated sector count aspart of S.M.A.R.T. attributes) are collected (e.g., by data collector151 or server 160) from a set of known failed disks associated with atarget storage system. A quantile distribution graph is generated (e.g.,by analysis module 153 or disk failure predictor 152) for the values inview of percentiles (e.g., 10%, 20%, . . . , 100%) of a number of knownfailed disks involved in the quantile distribution. Subsequently, whenvalues of the predetermined diagnostic parameter are collected from aset of target storage disks, the collected values are applied (e.g., bydisk failure predictor 152) to the quantile distribution graph todetermine their respective percentiles of the target disks. Eachpercentile represents a probability of the corresponding disk failure.The individual disk failure probabilities are then used (e.g., by diskfailure predictor 152) to calculate the probability of disk failures oftwo or more of the target disks.

FIG. 2 is a flow diagram illustrating a method for predicting diskfailures according to one embodiment of the invention. Method 200 may beperformed by processing logic which may include software, hardware, or acombination thereof. For example, method 200 may be performed bymanagement system 150 of FIG. 1. Referring to FIG. 2, at block 201,processing logic collects diagnostic parameters (e.g., at least aportion of S.M.A.R.T. attributes) from a set of historically knownfailed disks and known working disks of one or more storage systems(e.g., deduplicated backup storage systems). At block 202, processinglogic performs an analysis on the collected diagnostic parameters toidentify one or more of the diagnostic parameters as one or more diskfailure indicators that can be used to consistently or reliably indicatefuture disk failures. In one embodiment, processing logic utilizesquantile distribution techniques to select one or more of the diagnosticparameters as one or more disk failure indicators. At block 203,processing logic monitors and collects diagnostic parameters from one ormore target disks of a target storage system. At block 204, processinglogic predicts, using the selected disk failure indicator(s), theprobability of disk failures of the one or more disks of the targetstorage system based on the collected diagnostic parameters. The diskfailure indicator(s) can be utilized to predict the probability ofsingle-disk configuration or multi-disk configuration such as a RAIDconfiguration.

As described above, a disk failure indicator can be identified andselected from a set of diagnostic parameters, such as the S.M.A.R.T.attributes set forth below in the Appendix I or SCISI return codes setforth below in the Appendix II, collected from a set of known faileddisks and working disks, using quantile distribution analysis. Quantilesare points taken at regular intervals from the cumulative distributionfunction of a random variable. Dividing ordered data into q essentiallyequal-sized data subsets is the motivation for q-quantiles; thequantiles are the data values marking the boundaries between consecutivesubsets. Put another way, the k^(th) q-quantile for a random variable isthe value x such that the probability that the random variable will beless than x is at most k/q and the probability that the random variablewill be more than x is at most (q−k)/q=1−(k/q). There are q−1 of theq-quantiles, one for each integer k satisfying 0<k<q.

As described above, in order to predict future disk failures, a diskfailure indicator that can consistently or reliably indicate future diskfailures must be identified. According to one embodiment, certaindiagnostic parameters, which may be selected from at least some of theS.M.A.R.T. attributes, are collected from a set of previously knownfailed disks (e.g., q failed disks) and a set of known working disks(e.g., q working disks). For each of the collected diagnosticparameters, a quantile distribution representation (also referred to asa quantile distribution graph or curve) is generated.

For example, for a set of known working disks (e.g., a predeterminednumber of working disks), values of a particular diagnostic parameterare collected from the set of working disks (for example, via adiagnostic or monitoring tool such as S.M.A.R.T. tool):

-   -   Working Set=[10, 20, 30, 40, 50, 60, 70, 80, 90, 100]        In this example, it is assumed there are 10 disks in each set.        The values of the particular diagnostic parameter are stored in        an array and sorted according to a predetermined order, in this        example, an ascending order, from small to large.

Similarly, values of the same diagnostic parameter are collected fromthe set of known failed disks:

-   -   Failed Set=[10, 50, 100, 150, 200, 250, 300, 350,4 00, 450]        Similar to the working set, the values of the same diagnostic        parameter for the failed disks are stored in an array and sorted        according to the same order as the working set (e.g., ascending        order).

These two arrays are then plotted against the percentiles of number ofthe disks involved (e.g., 10 disks here) to generate the quantiledistribution representations or graphs for both the known failed disksand known working disks, as shown in FIG. 3A. In the example as shown inFIG. 3A, a first quantile distribution representation 301 is generatedrepresenting the known working disks while a second quantiledistribution representation 302 is generated representing the knownfailed disks. Referring to FIG. 3A, two quantile distributionrepresentations are then analyzed to determine the maximum or mostsignificant difference value between the two quantile distributionrepresentations 301-302. In the example as shown in FIG. 3A, the maximumdifference value 303 can be identified at the 100% location, where themaximum difference value is (450−100)=350.

Similar processes are performed for all of the diagnostic parametersthat have been selected as disk failure indicator candidates. Themaximum difference values associated with the diagnostic parameters arecompared with each other. In one embodiment, one of the diagnosticparameter associated with the largest difference value amongst all maybe selected as a disk failure indicator. In one embodiment, a distancebetween two quantile curves may be evaluated by calculating the areasize between the two curves. This takes into account all the points onthe curves rather than one specific point. The parameter that has thebiggest area size between two curve is the best disk failure indicator.

FIG. 3B is a diagram illustrating quantile distributions of variousdiagnostic parameters according to certain embodiments of the invention.Referring to FIG. 3B, for the purpose of illustration, the quantiledistribution representations are generated based on certain selectivediagnostic parameters, such as reallocated sector (EAS, with code 0x05),medium error, timeout, pending sector (0xC5) or uncorrectable sector(0xC6), connection error, data error, and other errors. However, otherdiagnostic parameters such as those S.M.A.R.T. attributes listed in theAppendix below can also be utilized to generate the quantiledistribution representations.

As shown in FIG. 3B, the maximum difference value in the RAS quantiledistribution representation is the largest amongst all of the quantiledistribution representations. Thus, in this example, RAS should beselected as the best disk failure indicator for the purpose ofindicating or predicting future disk failures. Note that the diagnosticparameters may be collected from a set of known or predetermined faileddisks and working disks of a particular operating environment, which maybe associated with a particular vendor, a customer or enterprise site,or a service provider, etc. Different configurations or diskmanufacturers may yield different values of diagnostic parameters, andtheir interpretation or definition may be different. Therefore, theselected disk failure indicator may be more accurate for indicating orpredicting future disk failures in the same or similar operatingenvironment in which those known failed dirks and working disks onceoperated.

FIG. 4 is a flow diagram illustrating a method for determining a diskfailure indicator according to one embodiment of the invention. Method400 may be performed by processing logic which may include software,hardware, or a combination thereof. For example, method 400 may beperformed by analysis module 153 of FIG. 1. Referring to FIG. 4, atblock 401, processing logic receives diagnostic parameters (e.g.,certain selected S.M.A.R.T. attributes) of a set of known orpredetermined failed disks and working disks associated with one or morestorage systems (e.g., backup storage systems). At block 402, for eachof the diagnostic parameters, processing logic generates a firstquantile distribution representation or graph for the set of workingdisks. At block 403, processing logic generates a second quantiledistribution representation for the set of failed disks. Once all of thequantile distribution representations for all of the diagnosticparameters have been generated, at block 404, the first and secondquantile representations of each diagnostic parameter are compared toselect one or more of the diagnostic parameters as one or more diskfailure indicators. Specifically, for each diagnostic parameter, amaximum or most significant difference value is determined, as describedabove, between the two quantile distribution representations thatrepresent a set of known working disks and a set of known failed disks,respectively. All the maximum difference values of all diagnosticparameters are compared to select the largest difference value amongstall to be a disk failure indicator. Alternatively, multiple diagnosticparameters whose maximum difference values greater than a predeterminedthreshold may be selected as multiple disk failure indicators.

According to one embodiment of the invention, once a disk failureindicator has been selected, the disk failure indicator may be used topredict the disk failure probability of a particular disk, referred toas a target disk of a target storage system. A value of a particulardiagnostic parameter (e.g., predetermined diagnostic parameterrepresenting the determined disk failure indicator) is obtained from atarget disk, for example, using a diagnostic tool such as a S.M.A.R.T.compatible monitoring tool. The value of the predetermined diagnosticparameter is then compared to a predetermined threshold that isassociated the predetermined diagnostic parameter to determine aprobability of disk failures for the target disk. For example, if thevalue of the predetermined diagnostic parameter is greater than thepredetermined threshold, the target disk is considered having a higherrisk of future disk failures; otherwise, the target disk is consideredas a working disk with a lower risk of disk failures. The value that isused to compare the predetermined threshold may be an averaged value ora mathematical representation or function of many values collected fromthe target disk over a predetermined period of time. In one embodiment,the predetermined threshold may be an optimal threshold that isdetermined based on the balances between the accuracy of detectingfuture disk failures and the risk of false positive detection of thetarget disk.

FIG. 5 is a flow diagram illustrating a method for predicting diskfailures of a single disk according to one embodiment of the invention.Method 500 may be performed by processing logic which may includesoftware, hardware, or a combination thereof. For example, method 400may be performed by disk failure predictor 152 of FIG. 1. Referring toFIG. 5, at block 501, processing logic selects a diagnostic parameter asa disk failure indicator based on operating status of a set of knownworking disks and known failed disks. The disk failure indicator may beselected using the techniques as described above. At block 502,processing logic determines an optimal threshold for the diagnosticparameter to categorize whether a particular disk (e.g., a target disk)has a higher risk of disk failures. At block 503, processing logicmonitors or receives values of the selected diagnostic parameter fromone or more disks of a target storage system. For each of the disks, atblock 504, the values of diagnostic parameter of the disks are comparedwith the optimal threshold. At block 505, processing logic categorizes adisk having a higher risk of disk failure if the value of the parameterof that disk is greater than the optimal threshold.

As illustrated above, the optimal threshold may determine the accuracyof the prediction. A too-high or too-low threshold may lead to missingcertain potential failed disks and/or false positive working disks. Inone embodiment, values of a predetermined diagnostic parameter(representing the selected disk failure indicator, such as thereallocated sector count as part of S.M.A.R.T. attributes) are collectedfrom a set of known failed disks and known working disks associated witha target storage system. A first and second quantile distribution graphsof the values of the predetermined diagnostic parameter for the knownfailed disks and known working disks are generated, respectively, inview of a range of values of the known failed disks. An optimalthreshold of the values of the predetermined diagnostic parameter isdetermined by comparing the first and second quantile distributiongraphs. In one embodiment, the optimal threshold is identified at alocation where the difference between the first and second quantiledistribution graphs reaches the maximum. The optimal threshold is thenutilized to indicate or predict whether a particular target disk mayhave a higher probability of disk failure, for example, by comparing thevalue of the predetermined diagnostic obtained from the target diskagainst the optimal threshold.

In one embodiment, all historical failed disks and working disks areanalyzed. The prediction system calculates how many failed disks wouldbe captured by a certain threshold (referred to as an accuracyprediction) and how many working disks would be captured by thisthreshold (referred to as false positive). A first curve or first set ofdata points of the accuracy prediction and a second curve or second setof data points of the false positive as a function of differentthresholds (e.g., threshold candidates) are generated, respectively. Thethreshold which has the biggest difference between the first and secondcurves or data points may be selected as the optimal threshold.

Assuming there are five failed disks with a selected diagnosticparameter (e.g., selected disk failure indicator), in this example RAS,having values of [30, 50, 50, 50, 60] (which may be sorted in apredetermined order such as ascending order). Five working disks withRAS=[10, 20, 30, 40, 40]. In one embodiment, the range of valuesobtained from the known failed disks may be utilized as a set ofthreshold candidates. For example, if the RAS threshold is set to be 30,the prediction algorithm will capture all the disks having RAS>=30 andregard them as impending failures. So it can capture 5 failed disks and3 working disks. So prediction accuracy=5, false positive=3. Likewise,if the threshold is set to be 40, prediction accuracy=4 and falsepositive=2. If the threshold is set to be 50, accuracy=4 and falsepositive=0. If the threshold is set to be 60, accuracy=1 and falsepositive=0, as shown in FIG. 6. From the curves as shown in FIG. 6, thelargest difference in values between the prediction accuracy curve 601and the false positive curve 602 appears at the threshold of 50. As aresult, the threshold of 50 may be selected as the optimal threshold forpredicting disk failures of a single target disk.

FIG. 7 is a flow diagram illustrating a method for determining anoptimal threshold for predicting disk failures according to oneembodiment of the invention. Method 700 may be performed by processinglogic which may include software, hardware, or a combination thereof.For example, method 700 may be performed by disk failure predictor 152of FIG. 1. Referring to FIG. 7, at block 701, processing logic collectsor receives values of a diagnostic parameter associated with a set ofknown failed disks and a set of working disks. The diagnostic parametermay have been selected as a disk failure indicator amongst manydiagnostic parameters such as those listed as part of S.M.A.R.T.attributes in the Appendix I or SCISI return codes in the Appendix IIbelow. The set of failed disks and working disks may be associated witha target storage system of which the future disk failures are to bepredicted. At block 702, processing logic generates a first set of datapoints representing a number of known failed disks that would have beencaptured or identified against different thresholds or thresholdcandidates. The range of the thresholds may be determined by the rangeof the values of the diagnostic parameters of the set of failed disks.At block 703, a second set of data points is generated, representing anumber of the known working disks that would have been captured againstthe different thresholds or threshold candidates. At block 704, thefirst set and second set of data points are compared in view of thedifferent thresholds. At block 705, one of the thresholds or thresholdcandidates may be selected as an optimal threshold that has the maximumdifference between the corresponding values of the first and second setsof data points.

According to another aspect of the invention, a selected disk failureindicator may also be used to predict the disk failure probability ofmultiple disks in a RAID configuration. According to one embodiment,values of a predetermined diagnostic parameter (representing theselected disk failure indicator, such as the reallocated sector count aspart of S.M.A.R.T. attributes) are collected from a set of known faileddisks associated with a target storage system. A quantile distributiongraph is generated for the values in view of percentiles (e.g., 10%,20%, . . . , 100%) of a number of known failed disks involved in thequantile distribution. Subsequently, when values of the predetermineddiagnostic parameter are collected from a set of target storage disks,the collected values are applied to the quantile distribution graph todetermine their respective percentiles of the known failed disks. Eachpercentile represents a probability of the corresponding disk failure.The individual disk failure probabilities are then used to calculate theprobability of disk failures of two or more of the target disks.

In one embodiment, values of a predetermined parameter (e.g., selecteddisk failure indicator such as a reallocated sector count) are collectedfrom a set of historically known failed disks. Assuming the number ofRAS value is N, the N values of the predetermined parameter are storedin an array and sorted according to a predetermined order such asascending order from smallest to biggest. A quantile distribution isgenerated based on the sorted numbers. For the purpose of illustratingonly, it is assumed the selected disk failure indicator is RASparameter. In one embodiment, the value of predetermined percentileinterval, in this example, 10 percentile=the RAS value at the [N*0.1]offset of the sorted array. Similarly, the value of 0.X=the RAS value atthe [N*0.x] offset of the sorted array.

For example, it is assumed there are 1000 RAS values corresponding to1000 known failed disks. After sorting of the 1000 RAS values, it isassumed the sorted array contains: 100, 105, 160 . . . (the arraylength=1000). If the value of 105 positioned in the sorted array happensto be 100 (array length*0.1=1000*0.1) the quanitle graph or curve at 0.1should be 105. Similarly, if value of 230 positioned in the sorted arrayhappens to be 200 (array length*0.2=1000*0.2), the quintile graph orcurve at 0.2 should be 230. It is assumed that for a particular set ofknown failed disks, its quantile distribution graph is shown in FIG. 8A.

Once the quantile distribution graph has been established, according toone embodiment, it can be used to predict multiple-disk failurescenario, for example, by calculating the probability of individual diskfailures and then determining the probability of two or more disks. Forexample, as shown in FIG. 8A, if a target disk's corresponding parametervalue is around 230, its individual probability of disk failures isabout 50%. Similarly, if a target disk's corresponding parameter valueis around 450, its individual probability of disk failures is about 60%,etc.

For the purpose of illustration, it is assumed there are four diskshaving the predetermined parameter of [11, 30, 110, 240]. By applyingthese numbers into the quantile distribution graph as shown in FIG. 8A,for example, by looking up these numbers in the Y axis of the graph, wecan obtain their respective failure probability numbers from thecorresponding X axis of the graph. In this example, value 11 of the disk1 falls between [0.2, 0.3]; value 30 of disk 2 falls between [0.3, 0.4];value 110 of disk 3 falls between [0.4, 0.5]; and value 240 of disk 4falls between [0.5, 0.6]. According to one embodiment, the probabilitywill be selected as an upper bound of the range (although a lower boundcan also be utilized). As a result, the individual failure probabilitiesof these four disks are [0.3, 0.4, 0.5, 0.6], respectively as shown inFIG. 8B.

From the individual failure probabilities of individual disks, theircorresponding probabilities of working disks can be derived asP(work)=1=P(fail), as shown in FIG. 8B. In a RAID configuration, theRAID group failure can be defined as two or more disk failures in thisexample:

-   -   P(RAID group failure)=P(disk failure        number>=2)=1−P(disk_failure_num=0)−P(disk_failure_num=1).    -   P        (disk_failure_num=0)=P(disk1_w)*P(disk2_w)*P(disk3_w)*P(disk4_w).    -   P        (disk_failure_num=1)=P(disk1_failure)*P(disk2_w)*P(disk3_w)*P(disk4_w)+P(disk1_w)*P(disk2_failure)*P(disk3_w)*P(disk4_w)+P(disk1_w)*P(disk2_w)*P(disk3_failure)*P(disk4_w)+P(disk1_w)*P(disk2_w)*P(disk3_w)*P(disk4_failure).

P(disk failure number=0) refers to the probability of no disk failure.P(disk failure number=1) refers to the probability of one disk failure.P(disk1_w), P(disk2_w), P(disk3_w), and P(disk4_w) refer to theprobabilities of working disk for disk 1 to disk 4, respectively.P(disk1_failure), P(disk2 _(—) failure), P(disk3 _(—) failure), andP(disk4 _(—) failure) refer to the probabilities of disk failure fordisk 1 to disk 4, respectively. Similarly, the probability of more thanany number of disks can also be calculated. According to one embodiment,there are two tunable parameters: 1) the number of fail disks to beprevented, where the default number here is >=2 and 2) the number ofdisks in the RAID group (in this example, the number of disks is 3).Both numbers are adjustable based on different requirements and systemsettings.

FIG. 9 is a flow diagram illustrating a method for predicting multipledisk failures according to one embodiment of the invention. Method 900can be performed by processing logic which may include software,hardware, or a combination thereof. For example, method 900 may beperformed by disk failure predictor 152 of FIG. 1. Referring to FIG. 9,at block 901, processing logic builds a quantile distribution graph of adiagnostic parameter (e.g., selected disk failure indicator) from a setof known failed disks. At block 902, processing logic determines diskfailure probability of disks by applying values of the diagnosticparameter collected from the target disks to the quantile distributiongraph. At block 903, the disk failure probability of multiple disks isthen calculated from the individual disk failure probability of theindividual target disks.

FIG. 10 is a block diagram illustrating a deduplication storage systemaccording to one embodiment of the invention. For example, deduplicationstorage system 1000 may be implemented as part of a deduplicationstorage system as described above, such as, for example, thededuplication storage system as shown in FIG. 1. In one embodiment,storage system 1000 may represent a file server (e.g., an appliance usedto provide network attached storage (NAS) capability), a block-basedstorage server (e.g., used to provide SAN capability), a unified storagedevice (e.g., one which combines NAS and SAN capabilities), a nearlinestorage device, a direct attached storage (DAS) device, a tape backupdevice, or essentially any other type of data storage device. Storagesystem 1000 may have a distributed architecture, or all of itscomponents may be integrated into a single unit. Storage system 1000 maybe implemented as part of an archive and/or backup system such as adeduplicating storage system available from EMC® Corporation ofHopkinton, Mass.

In one embodiment, storage system 1000 includes a deduplication engine1001 interfacing one or more clients 1014 with one or more storage units1010 storing metadata 1016 and data objects 1018. Clients 1014 may beany kinds of clients, such as, for example, a client application, backupsoftware, or a garbage collector, located locally or remotely over anetwork. A network may be any type of networks such as a local areanetwork (LAN), a wide area network (WAN) such as the Internet, acorporate intranet, a metropolitan area network (MAN), a storage areanetwork (SAN), a bus, or a combination thereof, wired and/or wireless.

Storage devices or units 1010 may be implemented locally (e.g., singlenode operating environment) or remotely (e.g., multi-node operatingenvironment) via an interconnect, which may be a bus and/or a network.In one embodiment, one of storage units 1010 operates as an activestorage to receive and store external or fresh user data, while theanother one of storage units 1010 operates as a target storage unit toperiodically archive data from the active storage unit according to anarchiving policy or scheme. Storage units 1010 may be, for example,conventional magnetic disks, optical disks such as CD-ROM or DVD basedstorage, magnetic tape storage, magneto-optical (MO) storage media,solid state disks, flash memory based devices, or any other type ofnon-volatile storage devices suitable for storing large volumes of data.Storage units 1010 may also be combinations of such devices. In the caseof disk storage media, the storage units 1010 may be organized into oneor more volumes of redundant array of inexpensive disks (RAID). Datastored in the storage units may be stored in a compressed form (e.g.,lossless compression: HUFFMAN coding, LEMPEL-ZIV WELCH coding; deltaencoding: a reference to a chunk plus a difference; etc.). In oneembodiment, different storage units may use different compressionmethods (e.g., main or active storage unit from other storage units, onestorage unit from another storage unit, etc.).

The metadata, such as metadata 1016, may be stored in at least some ofstorage units 1010, such that files can be accessed independent ofanother storage unit. Metadata of each storage unit includes enoughinformation to provide access to the files it contains. In oneembodiment, metadata may include fingerprints contained within dataobjects 1018, where a data object may represent a data chunk (alsoreferred to as a data segment), a compression region (CR) of one or moredata chunks, or a container of one or more CRs. Fingerprints are mappedto a particular data object via metadata 1016, enabling the system toidentify the location of the data object containing a chunk representedby a particular fingerprint. When an active storage unit fails, metadatacontained in another storage unit may be utilized to recover the activestorage unit. When one storage unit is unavailable (e.g., the storageunit has failed, or is being upgraded, etc.), the system remains up toprovide access to any file not stored in the failed storage unit. When afile is deleted, the metadata associated with the files in the system isupdated to reflect that the file has been deleted.

In one embodiment, the metadata information includes a file name, astorage unit identifier identifying a storage unit in which the chunksassociated with the file name are stored, reconstruction information forthe file using the chunks, and any other appropriate metadatainformation. In one embodiment, a copy of the metadata is stored on astorage unit for files stored on a storage unit so that files that arestored on the storage unit can be accessed using only the informationstored on the storage unit. In one embodiment, a main set of metadatainformation can be reconstructed by using information of other storageunits associated with the storage system in the event that the mainmetadata is lost, corrupted, damaged, etc. Metadata for a storage unitcan be reconstructed using metadata information stored on a main storageunit or other storage unit (e.g., replica storage unit). Metadatainformation further includes index information (e.g., locationinformation for chunks in storage units, identifying specific dataobjects).

In one embodiment, deduplication storage engine 1001 includes fileservice interface 1002, segmenter 1004, duplicate eliminator 1006, filesystem control 1008, and storage unit interface 1012. Deduplicationstorage engine 1001 receives a file or files (or data item(s)) via fileservice interface 1002, which may be part of a file system namespace1020 of a file system associated with the deduplication storage engine1001. The file system namespace 1020 refers to the way files areidentified and organized in the system. An example is to organize thefiles hierarchically into directories or folders, which may be managedby directory manager 1022. File service interface 1012 supports avariety of protocols, including a network file system (NFS), a commonInternet file system (CIFS), and a virtual tape library interface (VTL),etc.

The file(s) is/are processed by segmenter 1004 and file system control1008. Segmenter 1004, also referred to as a content store, breaks thefile(s) into variable-length chunks based on a variety of rules orconsiderations. For example, the file(s) may be broken into chunks byidentifying chunk boundaries using a content-based technique (e.g., afunction is calculated at various locations of a file, when the functionis equal to a value or when the value is a minimum, a maximum, or othervalue relative to other function values calculated for the file), anon-content-based technique (e.g., based on size of the chunk), or anyother appropriate technique. In one embodiment, a chunk is restricted toa minimum and/or maximum length, to a minimum or maximum number ofchunks per file, or any other appropriate limitation.

In one embodiment, file system control 1008, also referred to as a filesystem manager, processes information to indicate the chunk(s)association with a file. In some embodiments, a list of fingerprints isused to indicate chunk(s) associated with a file. File system control1008 passes chunk association information (e.g., representative datasuch as a fingerprint) to index 1024. Index 1024 is used to locatestored chunks in storage units 1010 via storage unit interface 1012.Duplicate eliminator 1006, also referred to as a segment store,identifies whether a newly received chunk has already been stored instorage units 1010. In the event that a chunk has already been stored instorage unit(s), a reference to the previously stored chunk is stored,for example, in a chunk tree associated with the file, instead ofstoring the newly received chunk. A chunk tree of a file may include oneor more nodes and each node represents or references one of thededuplicated chunks stored in storage units 1010 that make up the file.Chunks are then packed by a container manager (which may be implementedas part of storage unit interface 1012) into one or more storagecontainers stored in storage units 1010. The deduplicated chunks may befurther compressed into one or more CRs using a variation of compressionalgorithms, such as a Lempel-Ziv algorithm before being stored. Acontainer may contains one or more CRs and each CR may contain one ormore deduplicated chunks (also referred to deduplicated segments). Acontainer may further contain the metadata such as fingerprints, type ofthe data chunks, etc. that are associated with the data chunks storedtherein.

When a file is to be retrieved, file service interface 1002 isconfigured to communicate with file system control 1008 to identifyappropriate chunks stored in storage units 1010 via storage unitinterface 1012. Storage unit interface 1012 may be implemented as partof a container manager. File system control 1008 communicates (e.g., viasegmenter 1004) with index 1024 to locate appropriate chunks stored instorage units via storage unit interface 1012. Appropriate chunks areretrieved from the associated containers via the container manager andare used to construct the requested file. The file is provided viainterface 1002 in response to the request. In one embodiment, filesystem control 1008 utilizes a tree (e.g., a chunk tree obtained fromnamespace 1020) of content-based identifiers (e.g., fingerprints) toassociate a file with data chunks and their locations in storageunit(s). In the event that a chunk associated with a given file or filechanges, the content-based identifiers will change and the changes willripple from the bottom to the top of the tree associated with the fileefficiently since the appropriate content-based identifiers are easilyidentified using the tree structure. Note that some or all of thecomponents as shown as part of deduplication engine 1001 may beimplemented in software, hardware, or a combination thereof. Forexample, deduplication engine 1001 may be implemented in a form ofexecutable instructions that can be stored in a machine-readable storagemedium, where the instructions can be executed in a memory by aprocessor.

In one embodiment, storage system 1000 may be used as a tier of storagein a storage hierarchy that comprises other tiers of storage. One ormore tiers of storage in this hierarchy may utilize different kinds ofstorage devices and/or may be optimized for different characteristicssuch as random update performance. Files are periodically moved amongthe tiers based on data management policies to achieve a cost-effectivematch to the current storage requirements of the files. For example, afile may initially be stored in a tier of storage that offers highperformance for reads and writes. As the file ages, it may be moved intoa tier of storage according to one embodiment of the invention. Invarious embodiments, tiers include different storage technologies (e.g.,tape, hard drives, semiconductor-based memories, optical drives, etc.),different locations (e.g., local computer storage, local networkstorage, remote network storage, distributed storage, cloud storage,archive storage, vault storage, etc.), or any other appropriate storagefor a tiered data storage system.

Some portions of the preceding detailed descriptions have been presentedin terms of algorithms and symbolic representations of operations ondata bits within a computer memory. These algorithmic descriptions andrepresentations are the ways used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of operations leading to adesired result. The operations are those requiring physicalmanipulations of physical quantities.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the above discussion, itis appreciated that throughout the description, discussions utilizingterms such as those set forth in the claims below, refer to the actionand processes of a computer system, or similar electronic computingdevice, that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

Embodiments of the invention also relate to an apparatus for performingthe operations herein. Such a computer program is stored in anon-transitory computer readable medium. A machine-readable mediumincludes any mechanism for storing information in a form readable by amachine (e.g., a computer). For example, a machine-readable (e.g.,computer-readable) medium includes a machine (e.g., a computer) readablestorage medium (e.g., read only memory (“ROM”), random access memory(“RAM”), magnetic disk storage media, optical storage media, flashmemory devices).

The processes or methods depicted in the preceding figures may beperformed by processing logic that comprises hardware (e.g. circuitry,dedicated logic, etc.), software (e.g., embodied on a non-transitorycomputer readable medium), or a combination of both. Although theprocesses or methods are described above in terms of some sequentialoperations, it should be appreciated that some of the operationsdescribed may be performed in a different order. Moreover, someoperations may be performed in parallel rather than sequentially.

Embodiments of the present invention are not described with reference toany particular programming language. It will be appreciated that avariety of programming languages may be used to implement the teachingsof embodiments of the invention as described herein.

In the foregoing specification, embodiments of the invention have beendescribed with reference to specific exemplary embodiments thereof. Itwill be evident that various modifications may be made thereto withoutdeparting from the broader spirit and scope of the invention as setforth in the following claims. The specification and drawings are,accordingly, to be regarded in an illustrative sense rather than arestrictive sense.

APPENDIX I S.M.A.R.T. Attributes ID Hex Attribute Name Description 010x01 Read Error Rate (Vendor specific raw value.) Stores data related tothe rate of hardware read errors that occurred when reading data from adisk surface. The raw value has different structure for differentvendors and is often not meaningful as a decimal number. 02 0x02Throughput Overall (general) throughput performance of a hard diskdrive. If the value of Performance this attribute is decreasing there isa high probability that there is a problem with the disk. 03 0x03Spin-Up Time Average time of spindle spin up (from zero RPM to fullyoperational [milliseconds]). 04 0x04 Start/Stop Count A tally of spindlestart/stop cycles. The spindle turns on, and hence the count isincreased, both when the hard disk is turned on after having before beenturned entirely off (disconnected from power source) and when the harddisk returns from having previously been put to sleep mode. 05 0x05Reallocated Sectors Count of reallocated sectors. When the hard drivefinds a Count read/write/verification error, it marks that sector as“reallocated” and transfers data to a special reserved area (sparearea). This process is also known as remapping, and reallocated sectorsare called “remaps”. The raw value normally represents a count of thebad sectors that have been found and remapped. Thus, the higher theattribute value, the more sectors the drive has had to reallocate. Thisallows a drive with bad sectors to continue operation; however, a drivewhich has had any reallocations at all is significantly more likely tofail in the near future. While primarily used as a metric of the lifeexpectancy of the drive, this number also affects performance. As thecount of reallocated sectors increases, the read/write speed tends tobecome worse because the drive head is forced to seek to the reservedarea whenever a remap is accessed. If sequential access speed iscritical, the remapped sectors can be manually marked as bad blocks inthe file system in order to prevent their use. 06 0x06 Read ChannelMargin Margin of a channel while reading data. The function of thisattribute is not specified. 07 0x07 Seek Error Rate (Vendor specific rawvalue.) Rate of seek errors of the magnetic heads. If there is a partialfailure in the mechanical positioning system, then seek errors willarise. Such a failure may be due to numerous factors, such as damage toa servo, or thermal widening of the hard disk. The raw value hasdifferent structure for different vendors and is often not meaningful asa decimal number. 08 0x08 Seek Time Performance Average performance ofseek operations of the magnetic heads. If this attribute is decreasing,it is a sign of problems in the mechanical subsystem. 09 0x09 Power-OnHours Count of hours in power-on state. The raw value of this attributeshows total (POH) count of hours (or minutes, or seconds, depending onmanufacturer) in power-on state. On some pre-2005 drives, this raw valuemay advance erratically and/or “wrap around” (reset to zeroperiodically). 10 0x0A Spin Retry Count Count of retry of spin startattempts. This attribute stores a total count of the spin start attemptsto reach the fully operational speed (under the condition that the firstattempt was unsuccessful). An increase of this attribute value is a signof problems in the hard disk mechanical subsystem. 11 0x0B RecalibrationRetries or This attribute indicates the count that recalibration wasrequested (under the Calibration Retry Count condition that the firstattempt was unsuccessful). An increase of this attribute value is a signof problems in the hard disk mechanical subsystem. 12 0x0C Power CycleCount This attribute indicates the count of full hard disk power on/offcycles. 13 0x0D Soft Read Error Rate Uncorrected read errors reported tothe operating system. 180 0xB4 Unused Reserved Block “Pre-Fail”Attribute used at least in HP devices. Count Total 181 0xB5 Program FailCount Total number of Flash program operation failures since the drivewas Total or Non-4K deployed. Aligned Access Count Number of user dataaccesses (both reads and writes) where LBAs are not 4 KiB aligned (LBA %8 != 0) or where size is not modulus 4 KiB (block count != 8), assuminglogical block size (LBS) = 512 B 183 0xB7 SATA Downshift Error WesternDigital and Samsung attribute. Count or Runtime Bad (or) Block Seagate.184 0xB8 End-to-End error/ This attribute is a part of Hewlett-Packard'sSMART IV technology, as well IOEDC as part of other vendors' IO ErrorDetection and Correction schemas, and it contains a count of parityerrors which occur in the data path to the media via the drive's cacheRAM. 185 0xB9 Head Stability Western Digital attribute. 186 0xBA InducedOp-Vibration Western Digital attribute. Detection 187 0xBB ReportedUncorrectable The count of errors that could not be recovered usinghardware ECC (see Errors attribute 195). 188 0xBC Command Timeout Thecount of aborted operations due to HDD timeout. Normally this attributevalue should be equal to zero and if the value is far above zero, thenmost likely there will be some serious problems with power supply or anoxidized data cable. 189 0xBD High Fly Writes HDD producers implement aFly Height Monitor that attempts to provide additional protections forwrite operations by detecting when a recording head is flying outsideits normal operating range. If an unsafe fly height condition isencountered, the write process is stopped, and the information isrewritten or reallocated to a safe region of the hard drive. Thisattribute indicates the count of these errors detected over the lifetimeof the drive. This feature is implemented in most modern Seagatedrives^([1]) and some of Western Digital's drives, beginning with the WDEnterprise WDE18300 and WDE9180 Ultra2 SCSI hard drives, and will beincluded on all future WD Enterprise products. 190 0xBE AirflowTemperature Airflow temperature on Western Digital HDs (Same as temp.[C2], but (WDC) resp. Airflow current value is 50 less for some models.Marked as obsolete.) Temperature Celsius (HP) 190 0xBE Temperature Valueis equal to (100-temp. ° C.), allowing manufacturer to set a minimumDifference from 100 threshold which corresponds to a maximumtemperature. 191 0xBF G-sense Error Rate The count of errors resultingfrom externally induced shock & vibration. 192 0xC0 Power-off RetractCount of times the heads are loaded off the media. Heads can be unloadedCount or Emergency without actually powering off. Retract Cycle Count(Fujitsu)^([2,3]) 193 0xC1 Load Cycle Count or Count of load/unloadcycles into head landing zone position. Load/Unload Cycle The typicallifetime rating for laptop (2.5-in) hard drives is 300,000 to Count(Fujitsu) 600,000 load cycles. Some laptop drives are programmed tounload the heads whenever there has not been any activity for about fiveseconds. Many Linux installations write to the file system a few times aminute in the background. As a result, there may be 100 or more loadcycles per hour, and the load cycle rating may be exceeded in less thana year. 194 0xC2 Temperature resp. Current internal temperature.Temperature Celsius 195 0xC3 Hardware ECC (Vendor specific raw value.)The raw value has different structure for Recovered different vendorsand is often not meaningful as a decimal number. 196 0xC4 ReallocationEvent Count of remap operations. The raw value of this attribute showsthe total Count count of attempts to transfer data from reallocatedsectors to a spare area. Both successful & unsuccessful attempts arecounted. 197 0xC5 Current Pending Sector Count of “unstable” sectors(waiting to be remapped, because of Count unrecoverable read errors). Ifan unstable sector is subsequently read successfully, this value isdecreased and the sector is not remapped. Read errors on a sector willnot remap the sector immediately (since the correct value cannot be readand so the value to remap is not known, and also it might becomereadable later); instead, the drive firmware remembers that the sectorneeds to be remapped, and will remap it the next time it's written.However some drives will not immediately remap such sectors whenwritten; instead the drive will first attempt to write to the problemsector and if the write operation is successful then the sector will bemarked good (in this case, the “Reallocation Event Count” (0xC4) willnot be increased). This is a serious shortcoming, for if such a drivecontains marginal sectors that consistently fail only after some timehas passed following a successful write operation, then the drive willnever remap these problem sectors. 198 0xC6 Uncorrectable Sector Thetotal count of uncorrectable errors when reading/writing a sector. Arise Count or in the value of this attribute indicates defects of thedisk surface and/or Offline Uncorrectable problems in the mechanicalsubsystem. or Off-Line Scan Uncorrectable Sector Count 199 0xC7 UltraDMACRC Error The count of errors in data transfer via the interface cableas determined by Count ICRC (Interface Cyclic Redundancy Check). 2000xC8 Multi Zone Error Rate The count of errors found when writing asector. The higher the value, the worse the disk's mechanical conditionis. 200 0xC8 Write Error Rate The total count of errors when writing asector. (Fujitsu) 201 0xC9 Soft Read Error Rate or Count of off-trackerrors. TA Counter Detected 202 0xCA Data Address Mark Count of DataAddress Mark errors (or vendor-specific). errors or TA Counter Increased203 0xCB Run Out Cancel 204 0xCC Soft ECC Correction Count of errorscorrected by software ECC 205 0xCD Thermal Asperity Rate Count of errorsdue to high temperature. (TAR) 206 0xCE Flying Height Height of headsabove the disk surface. A flying height that's too low increases thechances of a head crash while a flying height that's too high increasesthe chances of a read/write error. 207 0xCF Spin High Current Amount ofsurge current used to spin up the drive. 208 0xD0 Spin Buzz Count ofbuzz routines needed to spin up the drive due to insufficient power. 2090xD1 Offline Seek Drive's seek performance during its internal tests.Performance 210 0xD2 Vibration During Write (found in a Maxtor 6B200M0200 GB and Maxtor 2R015H1 15 GB disks) 211 0xD3 Vibration During WriteVibration During Write 212 0xD4 Shock During Write Shock During Write220 0xDC Disk Shift Distance the disk has shifted relative to thespindle (usually due to shock or temperature). Unit of measure isunknown. 221 0xDD G-Sense Error Rate The count of errors resulting fromexternally induced shock & vibration. 222 0xDE Loaded Hours Time spentoperating under data load (movement of magnetic head armature) 223 0xDFLoad/Unload Retry Count of times head changes position. Count 224 0xE0Load Friction Resistance caused by friction in mechanical parts whileoperating. 225 0xE1 Load/Unload Cycle Total count of load cycles Count226 0xE2 Load ‘In’-time Total time of loading on the magnetic headsactuator (time not spent in parking area). 227 0xE3 Torque AmplificationCount of attempts to compensate for platter speed variations Count 2280xE4 Power-Off Retract The count of times the magnetic armature wasretracted automatically as a Cycle result of cutting power. 230 0xE6 GMRHead Amplitude Amplitude of “thrashing” (distance of repetitiveforward/reverse head motion) 230 0xE6 Drive Life Protection Currentstate of drive operation based upon the Life Curve Status 231 0xE7Temperature Drive Temperature 231 0xE7 SSD Life Left Indicates theapproximate SSD life left, in terms of program/erase cycles or Flashblocks currently available for use. 232 0xE8 Endurance Remaining Numberof physical erase cycles completed on the drive as a percentage of themaximum physical erase cycles the drive is designed to endure 232 0xE8Available Reserved Intel SSD reports the number of available reservedspace as a percentage of Space reserved space in a brand new SSD. 2330xE9 Power-On Hours Number of hours elapsed in the power-on state. 2330xE9 Media Wearout Intel SSD reports a normalized value of 100 (when theSSD is new) and Indicator declines to a minimum value of 1. It decreaseswhile the NAND erase cycles increase from 0 to the maximum-rated cycles.234 0xEA Average erase count Decoded as: byte 0-1-2 = average erasecount (big endian) and byte 3-4-5 = AND Maximum Erase max erase count(big endian) Count 235 0xEB Good Block Count decoded as: byte 0-1-2 =good block count (big endian) and byte 3-4 = AND System(Free)system(free) block count. Block Count 240 0xF0 Head Flying Hours Timewhile head is positioning 240 0xF0 Transfer Error Rate Count of timesthe link is reset during a data transfer. (Fujitsu) 241 0xF1 Total LBAsWritten Total count of LBAs written 242 0xF2 Total LBAs Read Total countof LBAs read. Some S.M.A.R.T. utilities will report a negative numberfor the raw value since in reality it has 48 bits rather than 32. 2500xFA Read Error Retry Rate Count of errors while reading from a disk 2540xFE Free Fall Protection Count of “Free Fall Events” detected

APPENDIX II SCSI Return Codes Category Key ASC ASCQ Error Condition NoSense 0 00 00 No error 0 5D 00 No sense - PFA threshold reached SoftError 1 01 00 Recovered Write error - no index 1 02 00 Recovered no seekcompletion 1 03 00 Recovered Write error - write fault 1 09 00 Trackfollowing error 1 0B 01 Temperature warning 1 0C 01 Recovered Writeerror with auto-realloc - reallocated 1 0C 03 Recovered Write error -recommend reassign 1 12 01 Recovered data without ECC using prey logicalblock ID 1 12 02 Recovered data with ECC using prev logical block ID 114 01 Recovered Record Not Found 1 16 00 Recovered Write error - DataSync Mark Error 1 16 01 Recovered Write error - Data Sync Error - datarewritten 1 16 02 Recovered Write error - Data Sync Error - recommendrewrite 1 16 03 Recovered Write error - Data Sync Error - dataauto-reallocated 1 16 04 Recovered Write error - Data Sync Error -recommend reassignment 1 17 00 Recovered data with no error correctionapplied 1 17 01 Recovered Read error - with retries 1 17 02 Recovereddata using positive offset 1 17 03 Recovered data using negative offset1 17 05 Recovered data using previous logical block ID 1 17 06 RecoveredRead error - without ECC, auto reallocated 1 17 07 Recovered Readerror - without ECC, recommend reassign 1 17 08 Recovered Read error -without ECC, recommend rewrite 1 17 09 Recovered Read error - withoutECC, data rewritten 1 18 00 Recovered Read error - with ECC 1 18 01Recovered data with ECC and retries 1 18 02 Recovered Read error - withECC, auto reallocated 1 18 05 Recovered Read error - with ECC, recommendreassign 1 18 06 Recovered data using ECC and offsets 1 18 07 RecoveredRead error - with ECC, data rewritten 1 1C 00 Defect List not found 1 1C01 Primary defect list not found 1 1C 02 Grown defect list not found 11F 00 Partial defect list transferred 1 44 00 Internal target failure 15D 00 PFA threshold reached Not Ready 2 04 00 Not Ready - Cause notreportable. 2 04 01 Not Ready - becoming ready 2 04 02 Not Ready - needinitialise command (start unit) 2 04 03 Not Ready - manual interventionrequired 2 04 04 Not Ready - format in progress 2 04 09 Not Ready -self-test in progress 2 31 00 Not Ready - medium format corrupted 2 3101 Not Ready - format command failed 2 35 02 Not Ready - enclosureservices unavailable 2 3A 00 Not Ready - medium not present 2 3A 01 NotReady - medium not present - tray closed 2 3A 02 Not Ready - medium notpresent - tray open 2 4C 00 Diagnostic Failure - config not loadedMedium Error 3 02 00 Medium Error - No Seek Complete 3 03 00 MediumError - write fault 3 10 00 Medium Error - ID CRC error 3 11 00 MediumError - unrecovered read error 3 11 01 Medium Error - read retriesexhausted 3 11 02 Medium Error - error too long to correct 3 11 04Medium Error - unrecovered read error - auto re-alloc failed 3 11 0BMedium Error - unrecovered read error - recommend reassign 3 14 01Medium Error - record not found 3 16 00 Medium Error - Data Sync Markerror 3 16 04 Medium Error - Data Sync Error - recommend reassign 3 1900 Medium Error - defect list error 3 19 01 Medium Error - defect listnot available 3 19 02 Medium Error - defect list error in primary list 319 03 Medium Error - defect list error in grown list 3 19 0E MediumError - fewer than 50% defect list copies 3 31 00 Medium Error - mediumformat corrupted 3 31 01 Medium Error - format command failed HardwareError 4 01 00 Hardware Error - no index or sector 4 02 00 HardwareError - no seek complete 4 03 00 Hardware Error - write fault 4 09 00Hardware Error - track following error 4 11 00 Hardware Error -unrecovered read error in reserved area 4 15 01 Hardware Error -Mechanical positioning error 4 16 00 Hardware Error - Data Sync Markerror in reserved area 4 19 00 Hardware Error - defect list error 4 1902 Hardware Error - defect list error in Primary List 4 19 03 HardwareError - defect list error in Grown List 4 31 00 Hardware Error -reassign failed 4 32 00 Hardware Error - no defect spare available 4 3501 Hardware Error - unsupported enclosure function 4 35 02 HardwareError - enclosure services unavailable 4 35 03 Hardware Error -enclosure services transfer failure 4 35 04 Hardware Error - enclosureservices refused 4 35 05 Hardware Error - enclosure services checksumerror 4 3E 03 Hardware Error - self-test failed 4 3E 04 Hardware Error -unable to update self-test 4 44 00 Hardware Error - internal targetfailure Illegal Request 5 1A 00 Illegal Request - parm list length error5 20 00 Illegal Request - invalid/unsupported command code 5 21 00Illegal Request - LBA out of range 5 24 00 Illegal Request - invalidfield in CDB (Command Descriptor Block) 5 25 00 Illegal Request -invalid LUN 5 26 00 Illegal Request - invalid fields in parm list 5 2601 Illegal Request - parameter not supported 5 26 02 Illegal Request -invalid parm value 5 26 03 Illegal Request - invalid field parameter -threshold parameter 5 26 04 Illegal Request - invalid release ofpersistent reservation 5 2C 00 Illegal Request - command sequence error5 35 01 Illegal Request - unsupported enclosure function 5 49 00 IllegalRequest - invalid message 5 53 00 Illegal Request - media load or ejectfailed 5 53 01 Illegal Request - unload tape failure 5 53 02 IllegalRequest - medium removal prevented 5 55 00 Illegal Request - systemresource failure 5 55 01 Illegal Request - system buffer full 5 55 04Illegal Request - Insufficient Registration Resources Unit Attention 628 00 Unit Attention - not-ready to ready transition (format complete) 629 00 Unit Attention - POR or device reset occurred 6 29 01 UnitAttention - POR occurred 6 29 02 Unit Attention - SCSI bus resetoccurred 6 29 03 Unit Attention - TARGET RESET occurred 6 29 04 UnitAttention - self-initiated-reset occurred 6 29 05 Unit Attention -transceiver mode change to SE 6 29 06 Unit Attention - transceiver modechange to LVD 6 2A 00 Unit Attention - parameters changed 6 2A 01 UnitAttention - mode parameters changed 6 2A 02 Unit Attention - log selectparms changed 6 2A 03 Unit Attention - Reservations pre-empted 6 2A 04Unit Attention - Reservations released 6 2A 05 Unit Attention -Registrations pre-empted 6 2F 00 Unit Attention - commands cleared byanother initiator 6 3F 00 Unit Attention - target operating conditionshave changed 6 3F 01 Unit Attention - microcode changed 6 3F 02 UnitAttention - changed operating definition 6 3F 03 Unit Attention -inquiry parameters changed 6 3F 05 Unit Attention - device identifierchanged 6 5D 00 Unit Attention - PFA threshold reached Write Protect 727 00 Write Protect - command not allowed Aborted B 00 00 AbortedCommand - no additional sense code Command B 1B 00 Aborted Command -sync data transfer error (extra ACK) B 25 00 Aborted Command -unsupported LUN B 3F 0F Aborted Command - echo buffer overwritten B 4300 Aborted Command - message reject error B 44 00 Aborted Command -internal target failure B 45 00 Aborted Command - Selection/Reselectionfailure B 47 00 Aborted Command - SCSI parity error B 48 00 AbortedCommand - initiator-detected error message received B 49 00 AbortedCommand - inappropriate/illegal message B 4B 00 Aborted Command - dataphase error B 4E 00 Aborted Command - overlapped commands attempted B 4F00 Aborted Command - due to loop initialisation Other E 1D 00Miscompare - during verify byte check operation x 05 00 Illegal requestx 06 00 Unit attention x 07 00 Data protect x 08 00 LUN communicationfailure x 08 01 LUN communication timeout x 08 02 LUN communicationparity error x 08 03 LUN communication CRC error x 09 00 vendor specificsense key x 09 01 servo fault x 09 04 head select fault x 0A 00 errorlog overflow x 0B 00 aborted command x 0C 00 write error x 0C 02 writeerror - auto-realloc failed x 0E 00 data miscompare x 12 00 address marknot found for ID field x 14 00 logical block not found x 15 00 randompositioning error x 15 01 mechanical positioning error x 15 02positioning error detected by read of medium x 27 00 write protected x29 00 POR or bus reset occurred x 31 01 format failed x 32 01 defectlist update error x 32 02 no spares available x 35 01 unspecifiedenclosure services failure x 37 00 parameter rounded x 3D 00 invalidbits in identify message x 3E 00 LUN not self-configured yet x 40 01DRAM parity error x 40 02 DRAM parity error x 42 00 power-on orself-test failure x 4C 00 LUN failed self-configuration x 5C 00 RPLstatus change x 5C 01 spindles synchronised x 5C 02 spindles notsynchronised x 65 00 voltage fault x ≧80 x Vendor specific x x ≧80Vendor specific

What is claimed is:
 1. A computer-implemented method of determining adisk failure indicator for predicting disk failures, the methodcomprising: receiving diagnostic parameters collected from a set ofknown working disks and a set of known failed disks of a storage system;for each of the diagnostic parameters, generating a first quantiledistribution representation for the set of known working disks, andgenerating a second quantile distribution representation for the set ofknown failed disks; and comparing the first quantile distributionrepresentation and the second quantile distribution representation ofeach of the diagnostic parameters to select one or more of thediagnostic parameters as one or more disk failure indicators forpredicting future disk failures.
 2. The method of claim 1, wherein thediagnostic parameters comprise at least one of reallocated sector count,medium error, timeout, pending sector count, uncorrectable sector count,connection error, and data error of the working and failed disks.
 3. Themethod of claim 1, wherein the selected diagnostic parameters comprise areallocated sector count.
 4. The method of claim 1, wherein comparingthe first quantile distribution representation and the second quantiledistribution representation comprises: identifying a maximum differencevalue between the first and second quantile distribution representationsfor each of the diagnostic parameters; and selecting one of thediagnostic parameters that has a largest maximum difference valueamongst the maximum difference values of all diagnostic parameters asthe disk failure indicator.
 5. The method of claim 1, wherein generatingthe first and second quantile distribution representation for each ofdiagnostic parameters comprise: storing values of the diagnosticparameter of the working disks in a first array and sorting data membersof the first array in a predetermined order; storing values of thediagnostic parameter of the failed disks in a second array and sortingdata members of the second array in the predetermined order; andplotting the first and second arrays in a first and second curvesagainst a set of fixed intervals representing a number of working disksor failed disks.
 6. The method of claim 5, wherein comparing the firstquantile distribution representation and the second quantiledistribution representation comprises: for each of diagnosticparameters, identifying a maximum difference value between the first andsecond curves; and selecting one of the diagnostic parameters, as a diskfailure indicator, that has a largest maximum difference value amongstthe maximum difference values of all diagnostic parameters.
 7. Themethod of claim 1, wherein the storage system is a deduplicated backupstorage system.
 8. A non-transitory machine-readable medium havinginstructions stored therein, which when executed by a processor, causethe processor to perform a method of determining a disk failureindicator for predicting disk failures, the method comprising: receivingdiagnostic parameters collected from a set of known working disks and aset of known failed disks of a storage system; for each of thediagnostic parameters, generating a first quantile distributionrepresentation for the set of known working disks, and generating asecond quantile distribution representation for the set of known faileddisks; and comparing the first quantile distribution representation andthe second quantile distribution representation of each of thediagnostic parameters to select one or more of the diagnostic parametersas one or more disk failure indicators for predicting future diskfailures.
 9. The non-transitory machine-readable medium of claim 8,wherein the diagnostic parameters comprise at least one of reallocatedsector count, medium error, timeout, pending sector count, uncorrectablesector count, connection error, and data error of the working and faileddisks.
 10. The non-transitory machine-readable medium of claim 8,wherein the selected diagnostic parameters comprise a reallocated sectorcount.
 11. The non-transitory machine-readable medium of claim 8,wherein comparing the first quantile distribution representation and thesecond quantile distribution representation comprises: identifying amaximum difference value between the first and second quantiledistribution representations for each of the diagnostic parameters; andselecting one of the diagnostic parameters that has a largest maximumdifference value amongst the maximum difference values of all diagnosticparameters as the disk failure indicator.
 12. The non-transitorymachine-readable medium of claim 8, wherein generating the first andsecond quantile distribution representation for each of diagnosticparameters comprise: storing values of the diagnostic parameter of theworking disks in a first array and sorting data members of the firstarray in a predetermined order; storing values of the diagnosticparameter of the failed disks in a second array and sorting data membersof the second array in the predetermined order; and plotting the firstand second arrays in a first and second curves against a set of fixedintervals representing a number of working disks or failed disks. 13.The non-transitory machine-readable medium of claim 12, whereincomparing the first quantile distribution representation and the secondquantile distribution representation comprises: for each of diagnosticparameters, identifying a maximum difference value between the first andsecond curves; and selecting one of the diagnostic parameters, as a diskfailure indicator, that has a largest maximum difference value amongstthe maximum difference values of all diagnostic parameters.
 14. Thenon-transitory machine-readable medium of claim 8, wherein the storagesystem is a deduplicated backup storage system.
 15. A data processingsystem, comprising: a processor; and a memory storing instructions,which when executed from the memory, cause the processor to perform amethod, the method including receiving diagnostic parameters collectedfrom a set of known working disks and a set of known failed disks of astorage system, for each of the diagnostic parameters, generating afirst quantile distribution representation for the set of known workingdisks, and generating a second quantile distribution representation forthe set of known failed disks, and comparing the first quantiledistribution representation and the second quantile distributionrepresentation of each of the diagnostic parameters to select one ormore of the diagnostic parameters as one or more disk failure indicatorsfor predicting future disk failures.
 16. The system of claim 15, whereinthe diagnostic parameters comprise at least one of reallocated sectorcount, medium error, timeout, pending sector count, uncorrectable sectorcount, connection error, and data error of the working and failed disks.17. The system of claim 15, wherein the selected diagnostic parameterscomprise a reallocated sector count.
 18. The system of claim 15, whereincomparing the first quantile distribution representation and the secondquantile distribution representation comprises: identifying a maximumdifference value between the first and second quantile distributionrepresentations for each of the diagnostic parameters; and selecting oneof the diagnostic parameters that has a largest maximum difference valueamongst the maximum difference values of all diagnostic parameters asthe disk failure indicator.
 19. The system of claim 15, whereingenerating the first and second quantile distribution representation foreach of diagnostic parameters comprise: storing values of the diagnosticparameter of the working disks in a first array and sorting data membersof the first array in a predetermined order; storing values of thediagnostic parameter of the failed disks in a second array and sortingdata members of the second array in the predetermined order; andplotting the first and second arrays in a first and second curvesagainst a set of fixed intervals representing a number of working disksor failed disks.
 20. The system of claim 19, wherein comparing the firstquantile distribution representation and the second quantiledistribution representation comprises: for each of diagnosticparameters, identifying a maximum difference value between the first andsecond curves; and selecting one of the diagnostic parameters, as a diskfailure indicator, that has a largest maximum difference value amongstthe maximum difference values of all diagnostic parameters.
 21. Thesystem of claim 15, wherein the storage system is a deduplicated backupstorage system.